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Abstract 

This research aimed to determine the relative efficiency of the types of school common to three administrations 
of the PISA (2003-2006-2009) by comparing their results, along with the activities that are to be performed for 
inefficient school types and changes in the efficiency value of the school types by year. The comparative analysis 
was based on information obtained from student questionnaires and cognitive skills tests. Turkish students 
completing the PISA in 2003, 2006, and 2009 comprised the population of the survey-based study. In the study, 
the data obtained from the PISA tests were analyzed using Data Envelopment Analysis (DEA). The results of 
the study indicated that the differences in quality among high schools are still prominent. The low economic, 
social, and cultural index of the elementary-school-level students, and vocational high school students' inability 
to allocate sufficient time for study outside of school hindered these schools' efficiency, as shown in the three 
administrations of the PISA. Consequently, student achievement was found to vary by year, and variables to be 
increased or decreased for improved student success have been identified. 
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Education is currently the most important 
component of economic and social development, 
and is in rapid and constant change worldwide. 
Education is one of the most effective tools of 
political, social, and cultural integration and 
management of changes (Milli Egitim Bakanhgi 
[MEB], 2005a). Educational institutions aim 
to cultivate individuals able to respond to the 
requirements of the constantly changing world 
(Kutlu, Dogan, & Karakaya, 2008). Students’ success 
in life depends on their attaining levels at which they 


are able to use the basic knowledge and skills they 
have learned during their school years (Berberoglu, 
2006). Looking at students’ levels of success in their 
school lives, we can make some deductions for 
the power of rivalry in countries’ future economy 
(Acar, 2008). By means of successful individuals 
will increase the workforce, which ensures the 
country’s development. Students’ academic success 
is considered an important representation of the 
effectiveness of the educational system in many 
countries. In the process of finding and improving 
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deficiencies in the educational system and its 
components, most decisions are made according 
to the findings obtained through objective 
instruments of measurement (MEB, 2010). 

One of the examinations used by the Ministry of 
National Education to evaluate student success 
is international, and compared the educational 
system with those of other countries: The Program 
for International Student Assessment (PISA), 
prepared by the Organization for Economic Co- 
Operation and Development (OECD). The PISA 
is a study carried out every three year to evaluate 
students’ knowledge in math, science, and reading 
skills, it evaluates how much knowledge 15-year- 
old students in the OECD and other participating 
countries have so that they can understand their 
place in the modern world (MEB, 2010). 

Turkey participated in PISA in 2003 for the first 
time. The 2003 PISA results demonstrated math 
literacy; the average score was 425. According 
to the experts, this score is under even basic 
proficiency level. Although important reforms were 
made in 2006, results from that year were the same. 
In 2009, Turkeys PISA score increased from 425 to 
446. From 2006, when the focus of field was science 
literacy, to the year 2009, Turkey has experienced 
a thirty score increase. In comparison to 2006 
PISA scores, literacy scores increased by 17 (MEB, 
2010). Although the scores increased, the position 
of Turkey in the international frame remained 
mostly unchanged. While Turkey stood between 
37 th and 44 th positions among 75 countries in 2006, 
in 2009 Turkey remained in 41 st and 43 rd positions 
among 65 countries. The fact that the nations 
standing did not change and that student success 
did not significantly improve should be analyzed by 
different statistics. 

Curricula have always been the focus of 
developments in education, requests for 
change, and innovation (Sonmez, 1991). Turkey 
implemented new curricula to meet the educational 
goals of the European Union in 2004 (Ak§it, 2007). 
In the revised curricula, the behaviorist approach 
was replaced by cognitive and constructivist 
approaches. According to constructivist learning 
theory, learning is not a passive process, but 
requires active student participation and is a 
continuous developmental process. 

The revised curriculum assumes that active student 
participation in lessons will facilitate permanent 
learning and therefore increase success. Therefore, 
one of the variables used in the study is the active 
participation of the students in their lessons. This 
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variable is defined as the opportunities teachers give 
students to speak their minds while discussing their 
coursework and it also includes the motivation the 
teacher provides for the students. 

Another variable of this study is related to the 
time students spend studying outside the school. 
This variable is measured by taking the average 
number of hours of private courses outside school 
hours (either in school, home, or any other place), 
independent study, and doing homework. Students 
active participation and time devoted to studying 
outside the school are the variables expected to 
contribute to increased levels of student success. 
Therefore, these variables are considered valuable 
for researchers. 

One of the changes that the MEBs implemented 
according to the improvements plan and 
government programs is a study aimed to streamline 
the curricula and course variety in secondary 
schools. With this objective, the number of types of 
schools in Turkey will be reduced to 15, consistent 
with the philosophy “single management, many 
programs.” (Giir & C^elik, 2009). 

Previous PISA studies conducted in Turkey have 
focused primarily on the PISA 2003 and/or 2006 
administrations and questionnaires, with the goal 
of discerning whether the impacts of mathematics/ 
science/reading achievement can be detected 
by comparison of different techniques within a 
three-year period (Akyiiz & Pala, 2010; Albayrak, 
2009; Anil, 2008, 2009; Aydin, Erdag, & Ta§, 2009; 
Aydogdu, Aydin, & Donmez, 2009; Berberoglu & 
Kalender, 2005; Boztun<;, 2010; Qif<;i, 2006; Demir, 
Depren, & K 1 I 19 , 2009; Depren, 2008; Dinner & 
Kola§in, 2009; Erba§, 2005; Guzel, 2006; Ozba§i, 
Demirta§li, Kumanda§, & Yal^in, 2010; Ozer, 2009; 
§a§mazel, 2006; Tomul & (Jelik, 2009; Usta, 2009; 
Uysal, Aydin, & Sarier, 2009; Yildirim, 2009) and the 
abroad (Afonso & Aubyn, 2005; Agasisti, in press; 
Cromley, 2009; Geske, Grinfelds, Dedze, & Zhang, 
2006; Lynn & Mikk, 2009; Mancebon, Calero, 
Choi, & Perez, 2010; Nonoyama, 2005; Perelman 
& Santin, 2008; Salasvelasco, 2006; Sutherland, 
Price, Joumard, & Nicq, 2007; Wolfram, 2005; Xu, 
2006). In Turkey, the research in which descriptive 
statistics are determined by multiple-variable 
variance analysis or multiple linear regression 
analysis have found that the levels of student 
success vary according to students’ socioeconomic 
status, their parents’ education level, and the type 
of school they attend. Furthermore, in the studies 
it was found out that there was no significant 
improvement in student success levels in these 
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years. In these studies, a single output variable 
(reading, math, or science) was used to limit the 
analysis. Furthermore, in the studies done carried 
out outside Turkey hierarchical linear modeling or 
data envelopment analysis were used to identify 
the factors efficient in changing student success, as 
well as changing student success itself, in different 
countries. Student success was has thus been seen 
to vary according the socioeconomic conditions 
and the parents’ levels of education. 

In the studies on PISA data in Turkey and abroad, 
parametric statistical methods and 2000, 2003 and/ 
or 2006 data were used. These studies commonly 
found that student success varied according to 
students’ socioeconomic status of the student and 
parents’ educational levels. In Turkey, a few studies 
(Demir & Depren, 2010; Depren, 2008) addressed 
DEA through PISA data, but many more of these 
studies (Afonso & Aubyn, 2005; Agasisti, in press; 
Cebada, Chaparro, & Gonzales, 2009; Ferrera, 
Cebada, Chaparro, & Gonzalez, 2011; Mancebon 
et al., 2010) have been conducted abroad than in 
Turkey. This research aims to contribute to this 
work, using PISA data with the non-parametric 
method DEA. This study differs from previous 
ones in looking at other studies in a long-term 
comparison from 2003 to 2009. Also this study 
analyzes scores for reading, math, and science 
simultaneously. Moreover, in these studies, the 
variables that are efficient in increasing student 
success are emphasized, but recommended 
increases or decreases in the variables are not 
clearly stated (Yun, Nakayama, & Tanino, 2004). 

1. It remains to be determined whether changes in 
available school types will affect student success. 
Likewise, it remains to be determined whether 
such changes will occur. The fact that student 
success is designated in terms of school years 
and the efficiency of educational investments are 
topics of growing interest worldwide. The results 
of the 2003 PISA, in which Turkey participated 
for the first time, triggered improvements to 
secondary school curricula. In this respect, 
the secondary school (6 th , 7 th , and 8 th grades) 
curricula introduced in 2006 and the effects of 
education investments on students’ academic 
success are two important issues to be kept in 
mind. Therefore, one must examine levels of 
success according to changes in school types and 
in particular through the comparison of primary 
school curricula introduced in 2006. This study 
addresses the following questions: For the 
periods of the 2003, 2006, and 2009 PISA, 


a. Which were the efficient school types? 

b. Which were the inefficient school types? 

c. Which school or schools form the reference set 
of efficient and inefficient school types? 

d. What should be required to the input levels and 
produce output levels for inefficient type/types 
of school? 

2. What is the efficiency value of each school type 
in each year? 

PISA results are a good predictor of student 
achievement, and the education system is of great 
importance for identifying problem points using 
long-term analysis of the PISA results. Studies using 
PISA data will be able to contribute to training 
policies and improving the quality and success rates 
in education and development. 

The studies which use PISA data, one of the 
studies used internationally will be able to bring a 
contribution to training policies and improving the 
quality and success in education, development and 
robust aspects of the education system 

This research is important because Turkish students’ 
achievement levels on the PISA are compared by 
type of school and year (2003, 2006, and 2009). In 
other words, it can be concluded that the education 
system has a serious problem considering Turkey’s 
PISA success in 2003, 2006, and 2009. For this 
reason, examining the PISA applications is crucial 
for determining problem points in the education 
system. 

The findings of this study will be significant for 
useful to measurement and evaluation experts in 
situations that require multiple inputs (i.e, students’ 
socio-economic status, the time students have 
available outside school for studying), outputs (i.e, 
students in math, science and reading achievement), 
and to those who wish to get better measurement 
results from their analytic work, particularly with 
studies analyzing multiple outputs as a single output. 
The fact that PISA applications include more than 
one input and output in different scales and that 
they benefit from DEA will better guide managers to 
understand the level of increase or decrease required 
in their inputs and outputs in inefficient units. It will 
also help us define the sources of the inefficiency by 
using the data from the 2009 PISA. This study will 
thus contribute to the field and will introduce the 
non-parametric method DEA to Turkey, where such 
studies are uncommon. 

Some of the concepts used in this research can be 
found below: 
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Input: The economic, social, and cultural status 
index, the active participation of students in classes, 
and the time students have available outside school 
for studying. All of these were among the items 
from the student questionnaires in the 2003, 2006, 
and 2009 PISA. 

Output: mathematics, science, and reading 
achievement scores from the 2003, 2006 and 2009 
PISA. 

Value Efficiency: The scores assigned to each 
student according to their success in math, science, 
and reading in relation the type of school attended. 

Decision-Making Unit: The type of school 
according to the scores calculated among the 
schools administering the 2003, 2006, and 2009 
PISA (Anatolian, science, general, Anatolian 
vocational, vocational high schools, and elementary 
school). 

Method 

Model of Research 

This study used a survey model to reveal the 
existing status of PISA achievement among 
Turkish students by year. 

Universe and Sampling 

The subjects of the study were Turkish students 
participating in the 2003, 2006, and 2009 
administrations of the PISA. This subject sample 
was determined by random sampling of students 
taken from the seven regions of Turkey in 2003, 
2006, and 2009. For all three PISA administration 
periods, z scores have been calculated in relation 
to the common school types included in the PISA 
for mathematics, reading, and science achievement 
scores, and the data beyond the +3 and -3 range 
have been omitted from the analyses. After the 
removal of extreme values, 4637, 4592, and 4412 
students were identified for 2003, 2006, and 2009 
respectively (MEB, 2005b, 2007, 2010). 

The Tools for Data Collection 

The cognitive skill tests and student questionnaires 
used in the 2003, 2006, and 2009 PISA were also 
used as data collection tools in this study. In the 
PISA, different types of items are used. Most of 
these consist of simple multiple-choice, yes/no, or 
Agree/Disagree choice questions. The remaining 
items are open-ended and require students to give 


their own short or long answers (MEB, 2010). 
The reliability and validity of PISA tests and 
questionnaires were determined using different 
approaches. For this reason, the views of experts, 
the average scores of the questions, and the answer 
codes were considered. The reliability coefficient 
was calculated with the Cronbach-alpha coefficient. 
For the 2003 PISA, this value was between 0.68 
and 0.93 (OECD, 2005); for the 2006 PISA, it was 
between 0.76 and 0.92 (OECD, 2007), confirming 
its reliability. 

Data Analysis 

The data from the 2003, 2006, and 2009 PISA were 
taken from the official PISA website (www.pisa. 
oecd.org). For the first goal in the study, DEA was 
used. The economic, social, and cultural status 
index, the active participation of the students, and 
available time for studying outside of school were 
chosen as inputs in the PISA applications, and they 
were defined according to three cognitive skills 
(math, science and reading achievement scores), 
which were posited as outputs. The choice of DEA 
was made for several reasons: It helps multiple- 
single output analysis and also determines the steps 
for making inefficient schools efficient. 

The second aim of the study was pursued using 
the Window Analysis, a DEA analysis developed 
by Charnes, Clark, Cooper, and Golany (1985), 
which is dependent on time. It is used to define 
how the efficiency value of time changes by year. 
In this analysis, for each decision-making unit, 
measured values are considered as if they were 
different decision-making units (Cooper, Seiford, 
& Zhu, 2004). The DEA is a linear-program-based 
method that aims to evaluate the performances 
of decision-making units at a time when different 
types of outputs and inputs make it difficult to make 
a comparison (Tarim, 2001; Thanassoulis, Portela, 
& Allen, 2004). The DEA is a non-parametric 
program developed to define the efficiency of 
decision-making units using linear programming 
(Aslankaraoglu, 2006). The analyses in this is 
study followed the steps of the DEA (Kecek, 2010; 
Ozyigit, 2000), as shown below. 

1. Choosing the Decision-making Units: In this 
study, the type of schools was the decision¬ 
making unit. 

2. Choosing the Input and Output: Among the 
items found in the 2003, 2006, and 2009 PISA 
student questionnaires, the items believed to 
help determine the efficacy of the program 
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initiated in 2006 were chosen, along with 
the input variables. For input variables, the 
economic, social, and cultural index, the active 
participation of students in classes, and their 
available time to study outside of school were 
chosen. As the output variable, for each PISA 
period, math, science and reading achievement 
scores were chosen. 

3. The Validity and the Availability of the Data 

4. Choosing the DEA Model and Measuring the 
Relative Efficiency: DEA models can be divided 
into two groups: One focuses on input and 
the one on output. In this research, under the 
assumption of constant return to scale, the CCR 
model was identified and used as the best way to 
ensure the best output (Bekta§, 2007). 

5. Efficiency Values: The type of school whose 
efficiency value is 1 is considered as efficient. The 
closer the value gets to 1, the more efficient the 
type of school will be. 

6. Reference/Control Group: Applying the same 
methods used by efficient groups for inefficient 
groups is assumed to raise the level of efficiency 
of the latter. 

7. Choosing Targets for Inefficient Units: 

Attainable targets are proposed to improve the 
performance of inefficient decision-making 
units (Aslankaraoglu, 2006; Aydagiin, 2003). 

8. Evaluation of the Results 

In this study, data obtained from the 2003, 2006, 
and 2009 PISA cognitive skills tests (mathematics, 
science, and reading) and student surveys were 
analyzed with SPSS (Version 17) and EMS programs 
vl.3.0 package (Scheel, 2000). 

Results 

In the three administrations of the PISA, the 
science high schools were found to be the most 
efficient school type. The Anatolian high schools 
were found efficient in the 2006 and 2009 PISA. 
Primary schools were the least efficient type of 
school in the three periods of the PISA. The low 
levels of active in-class participation and math 
scores of the Anatolian high school-level students 
lowered the efficiency of this school type in the 
2003 PISA. Furthermore, the low economic, social, 
and cultural index and math scores of students 
in the General, Anatolian vocational, vocational 
high schools, and elementary schools lowered the 
efficiency of these schools in the 2003 PISA. 


Efficiency Values by School Type According to 
the 2003 PISA, and Steps to Improve Efficiency 

The only efficient school type according to the 
2003 PISA was the science high schools. The 
school types, ranked by efficiency, are as follows: 
science, Anatolian, general, Anatolian vocational, 
vocational high, and primary schools. According 
to the results of current research, all inefficient 
school types should take the science high schools 
as a model. Relatively low math scores and low 
levels of student participation in the Anatolian high 
schools, along with the low ESCS scores, lowered 
the efficiency of these schools, while students in the 
other types of schools had lower math scores, which 
also lowered their efficiency. 

Efficiency Values by School Type According to 
the 2006 PISA and Steps to Improve Efficiency 

The science and Anatolian high school demonstrated 
100% efficiency. Except for the Anatolian vocational 
high schools, all of the inefficient schools used the 
science high schools as their model. Anatolian 
vocational high schools should take both Anatolian 
and science high schools as their model. The lack of 
available study time outside of school and low math 
scores of students in the vocational and Anatolian 
vocational high schools lowered the efficiency of 
these school types, causing these students to have 
lower values than the students in general high 
schools with relatively low science and math scores. 

The low ESCS values and low math scores of primary 
school students lowered the efficiency of their schools. 
Apart from the inefficient primary schools, all schools 
were given Anatolian high schools as well as Science 
high schools as models to emulate. We recommend 
that primary schools take both Anatolian high 
schools and Science high schools as their models. We 
also recommend that all school types, except for the 
Anatolian vocational high schools, take Anatolian 
high schools as their model, while primary schools 
should take science schools as their model. 

Efficiency Values by School Type According to the 
2009 PISA, and Steps to Improve Their Efficiency 

The lack of available study time outside of class 
and low math scores of students of general and 
vocational high schools lowered the efficiency 
of Anatolian vocational high schools. Moreover, 
students with low ESCS values and math scores 
in vocational high schools and primary schools 
lowered the efficiency of these school types 
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Changes in Efficiency By Year According to the 
PISA 

The efficiency values of primary schools and 
vocational high schools in 2006 decreased in 
2009. The general and Anatolian vocational high 
schools increased their efficiency values from 2006 
to 2009. According to the 2003 PISA, the school 
type showing the greatest increase in 2006 was the 
vocational high school. According to the 2006 PISA, 
the Anatolian vocational high schools showed the 
greatest increase in 2009 of all the school types. 

Discussion 

Student success varied annually. It remains to be 
determined which variables should be increased or 
decreased to improve student success. The findings 
gathered through the first analysis of 2003 PISA, 
in the studies of Berberoglu and Kalender (2005), 
and Berberoglu (2007); it has been suggested 
that general, vocational, and Anatolian high 
schools participating in the 2003 PISA showed 
low performance levels, with general high schools 
and primary schools in particular being below the 
international average and that science high schools 
and Anatolian high school showed consistently 
high performance levels. These findings parallel 
those of Depren (2008) for the best school types 
according to the 2003 PISA. (Jif<;i (2006) asserted 
that the students participating in the PISA 2003, are 
below the Turkish average, and the type of school 
affects student success. 

In their findings for the 2006 PISA, Demir et al. 
(2009) suggested that the type of school has an effect 
on students’ math scores, and the most efficient 
schools are the science high schools, followed by 
the Anatolian high schools. These findings parallel 
those of Albayrak (2009). 


Dinner and Kola§in (2009) found that the success 
of individual schools varies significantly in Turkey. 
A report announced by the Education Reform 
Initiative (Egitim Reformu Giri§imi [ERI], 2011) 
showed that schools differed in socioeconomic 
conditions. A study by the Ministry of National 
Education (MEB, 2010) about the PISA studies 
showed that the differences between the schools 
and the school types are evident, and the effect 
of socioeconomic condition on the schools is 
clear. Student success varies by socioeconomic 
conditions, which is also seen in studies carried out 
in Turkey internationally on the PISA. 

These findings suggest that the Ministry of National 
Education should improve the efficiency of the 
curriculum by aiming to decrease the curricula and 
number of school types nationwide if necessary. 
Equality of opportunity is not maintained under the 
current system, and the differences in quality affect 
students in high school entrance examinations. This 
study also suggests considering the socioeconomic 
conditions of students, some other precautions 
should be taken. 

Future research can be conducted with different 
output variables. A nationwide comparison may 
be done if the data are made available. Since the 
names of the schools attending EARGED in 2009 
weren’t disclosed and the legal requirements of 
the PISA prevented school questionnaires from 
being included in this research, some questions 
remain. If the data are made public, they can be 
used in the school questionnaires. The increase in 
the decision-making units would make it possible 
to use more output and input variables and to 
make the efficiency values more distinguishable. 
Therefore, researchers can study different output 
and input variables by increasing the number of 
decision-making units (for example, considering 
each participating school a decision-making unit). 
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